dbx_ingestion_monitoring #126
Conversation
| } | ||
| } | ||
| ], | ||
| "catalog": "chavdarbotev_std", |
There was a problem hiding this comment.
Should this say chavdar?
|
|
||
| sys.path.append("../../lib") | ||
|
|
||
| from dbx_ingestion_monitoring.common_ldp import * |
There was a problem hiding this comment.
Should this be called "dbx_ingestion_monitoring"? It sounds very much like https://github.com/databrickslabs/dbx to me
|
|
||
| # Configure logging | ||
| logging.basicConfig( | ||
| level=logging.INFO, |
There was a problem hiding this comment.
You should check the formatting of this file, I think might be a mix of 2-spaces and 4-spaces?
| import sys | ||
| import logging | ||
|
|
||
| sys.path.append("../../lib") |
There was a problem hiding this comment.
|
|
||
| sys.path.append("../../lib") | ||
|
|
||
| from dbx_ingestion_monitoring.common_ldp import * |
There was a problem hiding this comment.
Nit: import * is generally frowned upon
| @@ -0,0 +1,60 @@ | |||
| { | |||
There was a problem hiding this comment.
This folder is a bit odd to me, it is called jobs but has notebooks? Should this live in an src directory of one of the DABs projects?
| @@ -0,0 +1,24 @@ | |||
| # This job builds an inverted index mapping pipeline tags to pipeline IDs. | |||
There was a problem hiding this comment.
So this directory is included from both DABs projects, right? Maybe it should be wrapped into something like common/?
| @@ -0,0 +1,17 @@ | |||
| Copyright (2025) Databricks, Inc. | |||
|
|
|||
| This Software includes software developed at Databricks (https://www.databricks.com/) and its use is subject to the included LICENSE file. | |||
There was a problem hiding this comment.
Does this refer to /LICENSE? You should probably make a copy
2 participants
This directory contains common code and DABs to deploy observability ETL and dashboards for Databricks ingestion projects. The goal is to provide an example and a starting point for building ingestion observability across pipelines and datasets.
In particular, the package provides:
cloud_file_states, system tables and other."tier:T0;team:data,tier:T1") instead of maintaining lists of pipeline IDsCurrently Generic SDP pipelines and Lakeflow CDC Connector pipelines are supported.